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Introduction 


Identification, resolution, and avoidance of technical and programmatic issues are important for 
ensuring safe and successful space missions. 1 Although the importance of applying lessons learned to 
reduce risk is frequently stressed, there is little material available to help technical and management 
personnel research and document lessons learned. Collecting, researching, identifying, and documenting 

lessons learned that will be useful to current and future management and engineering personnel is not 

2 

always a straightforward task." This white paper presents lessons learned and best practices concerning the 
research and documentation of technical and organizational lessons learned. It is intended to enable 
organizations to initiate or improve lessons learned research and documentation efforts. 

The content of this white paper is based on four technical lessons learned projects conducted by the 
United Space Alliance (USA) Flight Design and Dynamics Department, in support of the NASA/JSC Flight 
Design and Dynamics Division. Each project published a report, titled as follows: 

GPS Lessons Learned From the ISS, Space Shuttle and X-38 3 

4 

Lessons Learned From Seven Space Shuttle Missions 

Space Shuttle Rendezvous and Proximity Operations Experience Report 5 

Navigation Technical History with Lessons Learned 6 

The four projects were different in availability of subject matter experts and primary source material, 
subject scope, and the level of effort required to produce the final report. However, generic lessons can be 
drawn from all of them. The best practices will be discussed by the phases of report research and 
development: 

Defining Report Requirements, Project Organization, and Schedule 
Collection and Analysis of Source Material 
Writing and Integrating the Report 
Review and Revision of the Report 
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Defining Report Requirements, Project Organization, and 
Schedule 

The scope of the report must be defined before research and writing can begin. Report scope will be 
dependent on customer requirements, schedule, budget, source documentation, subject matter experts, and 
personnel who will conduct the research and writing tasks. A project leader who has final authority to 
make editorial decisions should be appointed. Personnel who are qualified and available to conduct 
research and writing should be identified. Input from professional technical writers and editors, along with 
publications personnel, should be included in the planning process. 

Personnel assigned to lessons learned tasks must have an aptitude for 
research, writing, and creating documents that engage the reader. 

Contributors should be subject matter experts with a recognized and demonstrated proficiency for research 
and writing. If contributors do not possess these skills a significant amount of project time may be spent 
resolving report quality issues. Furthermore, subject matter experts that lack research and writing skills 
may not make sufficient progress, posing a risk to cost and schedule, as well as risk of not meeting 
requirements for report content. If a subject matter expert must be used whose research and writing ability 
is not known, a short research and writing assignment could be given them to identify any potential 
research and writing issues. Examples of good lessons learned research and writing could be provided to 
aid in skill development. Review of draft text early in the project by a technical writer or professional 
editor may also promote skill development. 

Define report audience, scope, outline, and format requirements before 
writing begins. The needs of current and future engineers and managers should be considered when 
defining the scope and format of the report. When deciding on format and content, determine who the 
audience is, and how the document should be written to effectively communicate. Different subject matter 
experts will have different views on experiences, lessons learned, and factors underlying success or failure. 
Report authors should consider placing raw data in an appendix so that readers may examine the evidence 
and form their own opinions, independent of the report authors. If not defined early in the project, too 
much time may be spent resolving report format and organization issues late in the project, when time is 
needed to resolve issues with content, style, or to perform additional research. A professional writer and 
editor can provide useful advice early in a project when defining report format and organization. 

Conduct preliminary research on the topic to help determine report scope. 

Authors should have some familiarity with technical and programmatic history to assist the customer in 
defining report scope and to effectively assess the value of source documentation and interviews later in the 
project. In addition to communicating with the report customer, meetings with subject matter experts are 
helpful in defining scope, format, and locating sources of documents and other subject matter experts. 
Seeking advice from external subject matter experts before research and writing begins establishes a 
relationship that will include them in the research, development, and subsequent review process. 

A lessons learned report is not a requirements document. Some subject matter 
experts may be apprehensive about document content due to the impact it could have on future 
programmatic decisions. It is important to state that the document is for informational purposes and that it 
is not a set of requirements for current or future flight programs. 

Document lessons learned from projects that successfully mitigated 
technical and programmatic risk. Lessons learned efforts are most often motivated by 
challenging projects, systems performance issues, or mishaps. In such projects lessons learned may be a 
frequent topic of discussion among personnel working in projects that have encountered technical and 
organizational challenges, making the task of collecting potential lessons learned straightforward. 
However, much can also be learned from projects that meet or exceed initial expectations. Investigators 
should seek the reasons behind project success. 
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Define multiple lessons learned efforts, each having a narrow scope. Projects 
with a smaller scope require fewer contributors and are easier to lead and coordinate than large projects. A 
well defined research and documentation project with a narrow scope will pose less risk to cost and 
schedule than a project with a broad scope. It is easier to get subject matter experts to review a small 
document than a large one. 

Consider researching and documenting lessons learned in a task that is 
separate from the research and documentation of technical history or 
process description. Technical history and process description is useful, but may require more 
work than lessons learned. There is also additional risk of a gradual increase in project scope. A document 
of lessons learned versus a historical document may require a different approach to research and writing. 
Reports with a broad scope will require multiple authors. This can result in differing levels of detail in 
research and writing styles during development. Resolving issues with differences in style and content can 
lead to late and extensive changes, further compressing project schedule. 

Whenever possible, assign only one author and researcher to a report 
section. Assigning multiple personnel to one section can result in diffusion of responsibility, prose 
that does not flow, organizational conflicts within a section, and redundant material. If a large section of a 
report must be assigned to multiple individuals, one person should be identified who has editorial control 
over that section. Other contributors assigned to that section should act in a supporting role. Clearly 
defined tasks should be assigned to supporting contributors. Each writer should read through the whole 
section to understand it before working with an editor. This is particularly important if more than one 
author has worked on a section. 

Develop a realistic schedule for editorial reviews and feedback from 
external subject matter experts. Reviews by editors and subject matter experts outside the 
project are critical to ensure accuracy and adequate coverage of the subject matter. However, such reviews 
take time. Project planning should reflect the possibility that reviewers have multiple tasks to perform that 
have higher priorities than the review. A reviewer may not be able to devote full attention to a review to 
meet an overly optimistic schedule. Turnaround on reviews of draft reports can be lengthy and adequate 
time must be scheduled for proper consideration. However, the review phase should not be longer than two 
or three months. Longer time periods, or an open ended schedule, may prompt subject matter experts to 
give the review a low priority, increasing the risk that feedback will not be provided. Some reviewers may 
want more time, as reviews can be difficult to conduct due to frequent interruptions in the work 
environment. The review phase should be scheduled well ahead of time to permit subject matter experts to 
adjust their schedules. This helps ensure participation by an appropriate cross section of personnel. The 
review process should start early in the project and the review phase should end about four weeks before 
the final version is due for publication. This allows for enough time to perform additional changes as well 
as quality checks on format, grammar, spelling, and content accuracy. 

Track budgeted versus actual hours of all personnel and organizations 
supporting the project and ensure that the level of work required can be 
accomplished with the available budget. The impact of gradual increases in project 
scope on available remaining budget and schedule may not always be obvious to contributors and project 
leadership. Continuous tracking of hours worked and correlation of worked hours with task content can 
help the project leader avoid budget and schedule surprises. Such situational awareness can enable project 
leadership to take early action to re-scope the project, or assign priorities to research tasks and issue 
resolution to ensure that document requirements are met within available schedule and budget. The project 
leader should be familiar with the level of work that is driven by the report scope, suggested improvements, 
and formatting to avoid negative impacts on employee work load, cost, and schedule. 

Determine what software will be used and identify potential compatibility 
issues. Differing versions of word processing software, equation editors, and portable document 
software can lead to a variety of unique technical and format problems that will introduce delays into a 
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schedule. These issues may also require special expertise and attention from professional editors and 
information technology specialists. 
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Collection and Analysis of Source Material 


Collecting and examining historical documentation is critical to the accuracy and content of the report, 
as much knowledge is forgotten over time. The scope, accuracy and applicability of historical 
documentation vary greatly. Interviews with subject matter experts can help identify critical documents 
and avenues of exploration, that will help in prioritizing topics and make efficient use of time. Source 
material should be organized in a fashion that facilitates easy retrieval in the future. More material may be 
collected than authors have time to examine. 

Useful source documentation may be found just about anywhere. The most 
valuable and time saving source documentation are reports written in complete sentences and paragraphs 
that were created at the time the lessons were learned. These documents and others of lesser value, such as 
presentation charts, may be in filing cabinets, informal and formal libraries, internal electronic databases, 
and even on the internet. Subject matter experts who have participated in investigations of anomalous 
system performance may have accumulated large amounts of primary source material. This material may 
include white papers, informal memos, and presentations representing the technical expertise and 
observations of many team members. Large bodies of primary source material may already have been 
collected by other individuals and projects. Use of such material will save a considerable amount of time. 
Publicly available program histories can also provide useful background information and may reference 
useful source material that could be obtained. Primary source documents may be placed on compact disks 
and delivered to the report customer along with the final report. This will permit future researchers access 
to original source documents in a timely manner. 

Conduct interviews of subject matter experts. One-on-one interviews can be 
instrumental for locating source material, collecting observations, and gaining understanding of how 
organizational politics and personalities played a role in the outcome of a project. While oral history is 
important for identifying the key players and political climate within a project, key details can be forgotten 
and stories can change over time. Personnel who joined a project long after it was initiated may not be 
fully familiar with the rationale behind early programmatic decisions. This may have to be taken into 
account when evaluating their observations. While interviews are valuable, the optimal source of lessons 
learned is usually documentation created at the time the lessons were learned. 

Explain to subject matter experts the goals, process, and value of the 
lessons learned research project. Creating a successful lessons learned report requires 
participation of subject matter experts (management and engineering) from various disciplines that may 
have busy schedules and have never before participated in such an activity. Some personnel may be eager 
to participate as they have learned the value of documenting experiences and lessons learned. Others may 
be uneasy about sharing their insight or knowledge. Projects that encountered difficulties can be a rich 
source of material, but some personnel may be reluctant to discuss past challenges, particularly if questions 
of human error or errors in judgment may arise. Collection, discussion, and preservation of experiences 
and lessons learned can be a controversial exercise. Understanding the purpose, goals, characteristics, and 
steps of the research project may help smooth relationships and encourage discussion between participants. 

Ensure capture and preservation of lessons learned by clearly documenting 
them during a project and at project completion. Documentation and preservation of 
lessons learned and experiences should be conducted both during a project and immediately after 
completion. Memories fade, personnel change jobs, and detailed accounts of lessons and experiences are 
quickly forgotten if not written down. Documentation and preservation should be defined as steps in a 
project plan before a project is initiated. Early in a project, management and technical personnel are most 
motivated to preserve lessons learned and highlight factors behind successes and failures. As a program 
progresses, there may be less motivation to formally document lessons learned. This can complicate future 
historical research. 
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Writing and Integrating the Report 


This phase involves analysis of source material, identification of lessons learned, best practices, and 
observations. Text is written and illustrations are obtained or created. Questions that arise during research 
and writing should be recorded for preservation and later research. Consultations with subject matter 
experts may be required to resolve some issues before the formal review process begins. Contributors or 
publications personnel integrate text and illustrations into a draft or modify an existing draft based on 
reviewer inputs. 

Adhere to the facts. Evaluation of projects that encountered difficulties involves discussion of 
controversial and sensitive topics. Authors should be careful to adhere to the facts and avoid assigning 
culpability to individuals and organizations. Negative personal comments about people and organizations 
should be kept out of the report. There may be some valid lessons and experiences that are too sensitive to 
be widely distributed. An individual or a team researching and documenting lessons learned is not an 
independent mishap investigation board. 

Strive to be objective. Report authors should be open-minded and consider all opinions and 
evidence provided by subject matter experts and documentation. When discussing problems it is important 
to be balanced and fair. For example, the GPS receiver on the Shuttle was a combat proven device, built by 
the thousands, which met the requirements of the original military customer. The source of GPS 
performance problems was not with a poor quality product, but rather using a software intensive device in 
an application for which it was not originally designed. Furthermore, decisions concerning organization, 
schedule, requirements, level of testing, assumptions driving project requirements, etc., were made by the 
Shuttle Program and supporting contractors. The GPS receiver vendor was not responsible for these 
decisions. These points were made in the GPS lessons learned report . 3 

Clearly document lessons and experiences so that they will be easy to 
understand and apply. Some recorded lessons learned in historical documentation are written in 
such a technical fashion that they are difficult for someone years later to understand and apply. Even 
technical lessons have at their core a fundamental problem that is common across many technical 
disciplines, such as failure to communicate or follow proper procedures. Lessons and experiences should 
be written so that they can be universally understood and applied by both management and engineering 
personnel from various technical disciplines. 

Include high-level background material to place lessons in context for the 
reader. Readers will not have the same knowledge of the topic as the subject matter experts and 
document authors. A factual history of the project is useful for helping the reader understand the lessons 
and experiences. Of particular importance is the rationale behind budget, schedule, policy, and 
organizational decisions that influenced the outcome of the project. 

Avoid duplicating technique, process, or architecture details that already 
exist in Other documents. The focus of a lessons learned report should be the lessons learned 
and experiences that are important to preserve. Some description of systems architecture, operational and 
algorithmic techniques, and processes may be necessary to give context to the lessons learned and 
experiences. Flowever, too much effort expended on these topics distracts from researching and 
documenting lessons learned. Sources can be provided, either in bibliographic form or the source 
documents themselves on a compact disk, for the reader who is interested in more detail. An annotated 
bibliography can help a reader identify references of interest. 

Report contributors should adhere to the customer's defined scope for the 
report, unless the scope is formally changed. Requirements creep can occur in report 
development as easily as it occurs in software development. Without specific requirements for what the 
report should address, the scope can increase as research progresses. This may lead to major formatting 
changes and significant addition of new material, all leading to compressed schedules and potential for 
missing project milestones, in addition to budget problems. Project leadership should discuss potential 
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changes in scope with the report customer and subject matter experts to determine if the proposed changes 
add value to the report. A close relationship with the report customer permits frequent discussion of project 
status and timely resolution of requirements and scope issues. 

Establish central file locations accessible to all contributors and clearly 
define content for each folder. Projects with more than one contributor will require close 
coordination. Written material and research notes will have to be shared and individual progress regularly 
communicated. The status and method of configuration control for the current working copies must be 
understood by all contributors. For folders or report sections that change often (i.e. references, document 
inventory), a list of items to be added is helpful to track what has and has not been incorporated. The 
contributors need to be informed of major changes by another person to their assigned sections in order to 
stay abreast of the current status of their work. This will avoid confusion when addressing editorial 
comments later in the project. This is particularly true in the event that one or more contributors are not 
available at critical times during the project. 

If possible, provide original files to graphics and publications personnel. 

Clean and sharp graphics and illustrations add value to a lessons learned report. However, a significant 
amount of time can be spent improving graphics and illustrations obtained from older internal documents 
through scanning and photocopying. Working with original files can speed up the process. In addition, 
providing graphics personnel with the original source of the illustration can be helpful. Using graphics 
already resident in current internal documents can save time as publications personnel may already have 
access to the original in electronic form. 
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Review and Revision of the Report 


Review and revision is performed once a draft has been assembled. Multiple review and draft update 
cycles may occur depending on the size of the report and the level of review needed. Feedback from 
subject matter experts can be obtained on an individual basis or in meetings held to discuss the draft. 
Written comments, with rationale for the comments explained using complete sentences, have been found 
to be the most helpful to report authors. 

Conduct a review with subject matter experts that did not contribute to the 
writing of the report. Reviews by subject matter experts are critical to ensure accuracy, coverage 
of important topics, and to promote sharing of and critical thinking about the experiences. Quick, 
insightful, and accurate reviews of draft reports by external subject matter experts provides contributing 
authors more time to resolve issues and perform additional research. Reviewers may also provide feedback 
for improving legibility of a report. The first draft could be presented to two or three key personnel 
(management and technical) for initial review. After inputs are received and changes are made, a review 
process with a larger group of subject matter experts can be initiated. Tasks and reviews can be delegated 
to smaller groups of people with relevant experience. However, it is also good to have an outside opinion 
on lessons and observations produced by small sub-teams. It is important for reviewers to understand that a 
draft is not a final product and that all material in the draft is subject to change. While meetings to discuss 
a draft report were found to be useful, the most valuable input was obtained from redlined copies or emails 
containing comments on the drafts. Far more lessons and experiences were harvested in this fashion during 
the review process than from oral interviews during the initial research phase or from group review 
discussions. The authors should be open to everyone's input. Showing flexibility and a willingness to 
make necessary changes to the draft will encourage participants to contribute. 

Use a draft as a means of getting subject matter experts to provide input. 

When asked to participate or provide examples of experiences and lessons learned, some subject matter 
experts may have difficulty providing input. The role of the primary author is to initiate momentum and 
glean information and history from subject matter experts. Distributing a draft lessons learned presentation 
or report for critique was found to be useful in provoking thought, analysis, and the sharing of experiences 
among management and technical personnel. A draft is also a vehicle for igniting creative thinking and 
discussion among subject matter experts of items that might not have surfaced during the original research 
phase. Reviews can help identify gaps in report coverage and lead authors to areas where more 
documentation needs to be collected and analyzed. 

A review meeting is useful if the subject matter experts have reviewed the 
document before the meeting. Multiple document review meetings at a regular time and 
location can be held. There should be enough time between meetings to permit report authors to make 
necessary changes, acquire and study additional historical material, and review the new draft before 
distribution to subject matter experts for review. Reviews should consist of a small group of key subject 
matter experts (preferably less than 10). Small groups enable meeting attendees to reach a quick consensus 
and make it easier for all attendees to participate in the discussion. 

Deliberation and consensus building is a process that takes time. While one 
person or a small group of people may serve as authors, in reality the quality of a report depends on the 
input from and deliberation between subject matter experts. Different people will view the project from 
different perspectives. Management and technical personnel may have different views on a project and it 
may be beneficial to have varying views in the report. Some participants may have a more accurate picture 
of events and observed things that the authors did not detect in their original research. There may also be 
differences in interpretation of events between individuals. Not all observations and opinions may be 
suitable for preservation in a report. Subject matter experts may not agree on everything. Document 
authors should strive for a set of lessons learned and experiences that best represent the subject addressed in 
the report and that would be most useful to management and technical personnel working on future 
projects. 



Assign priorities to suggestions for expanded report scope. Reviews by editors 
and external subject matter experts may result in suggestions for additional topics to be covered. Such 
suggestions may enhance the value of the report. However, additional material may require a significant 
amount of time and effort to research and add to the report. Priorities should be assigned to potential scope 
expansions to ensure that workload, cost, and schedule are not negatively impacted. Once priorities and 
responsible contributors have been assigned, a list of additional work and research to be performed should 
be maintained and status should be tracked. This enables quick assessment of report progress by 
management. 
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Conclusion 


Identification, resolution, and avoidance of technical and programmatic issues are important for 
reducing risk in spaceflight programs. Documented lessons learned, experiences, and best practices enable 
program personnel to mitigate recurring technical and programmatic issues that negatively impact cost, 
schedule, system performance, mission success, and safety. Such documentation plays a key role in 
preserving NASA’s corporate knowledge for new and continuing programs. It is also a part of the career 
learning process and technical skill development of engineers and managers. All four lessons learned 
reports produced by the USA Flight Design and Dynamics Department provided timely lessons that have 
proven to be applicable to other projects and flight programs, both current and in development. 

Research and documentation of lessons learned and experiences requires personnel with an aptitude for 
research and writing. In addition, contributing authors must possess technical expertise and an 
understanding of the programmatic aspects of space flight projects and programs. Document scope and 
format requirements should be defined before written material is developed. Interviews with subject matter 
experts are useful when defining scope, as well as for researching lessons learned. Project schedule should 
provide enough time for editorial reviews and feedback from external subject matter experts and the 
production of interim drafts. Lessons and experiences should be clearly documented so that they will be 
easy to understand and apply. High-level background material is useful to place the lessons in context for 
the reader. Lessons learned are most efficiently and effectively captured during and at the end of a project, 
and should be a natural outcome of every project. 
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